50 research outputs found
Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations
We introduce the concept of numerical Gaussian processes, which we define as
Gaussian processes with covariance functions resulting from temporal
discretization of time-dependent partial differential equations. Numerical
Gaussian processes, by construction, are designed to deal with cases where: (1)
all we observe are noisy data on black-box initial conditions, and (2) we are
interested in quantifying the uncertainty associated with such noisy data in
our solutions to time-dependent partial differential equations. Our method
circumvents the need for spatial discretization of the differential operators
by proper placement of Gaussian process priors. This is an attempt to construct
structured and data-efficient learning machines, which are explicitly informed
by the underlying physics that possibly generated the observed data. The
effectiveness of the proposed approach is demonstrated through several
benchmark problems involving linear and nonlinear time-dependent operators. In
all examples, we are able to recover accurate approximations of the latent
solutions, and consistently propagate uncertainty, even in cases involving very
long time integration
Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations
We introduce physics informed neural networks -- neural networks that are
trained to solve supervised learning tasks while respecting any given law of
physics described by general nonlinear partial differential equations. In this
second part of our two-part treatise, we focus on the problem of data-driven
discovery of partial differential equations. Depending on whether the available
data is scattered in space-time or arranged in fixed temporal snapshots, we
introduce two main classes of algorithms, namely continuous time and discrete
time models. The effectiveness of our approach is demonstrated using a wide
range of benchmark problems in mathematical physics, including conservation
laws, incompressible fluid flow, and the propagation of nonlinear shallow-water
waves
Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations
We introduce physics informed neural networks -- neural networks that are
trained to solve supervised learning tasks while respecting any given law of
physics described by general nonlinear partial differential equations. In this
two part treatise, we present our developments in the context of solving two
main classes of problems: data-driven solution and data-driven discovery of
partial differential equations. Depending on the nature and arrangement of the
available data, we devise two distinct classes of algorithms, namely continuous
time and discrete time models. The resulting neural networks form a new class
of data-efficient universal function approximators that naturally encode any
underlying physical laws as prior information. In this first part, we
demonstrate how these networks can be used to infer solutions to partial
differential equations, and obtain physics-informed surrogate models that are
fully differentiable with respect to all input coordinates and free parameters
Inferring solutions of differential equations using noisy multi-fidelity data
For more than two centuries, solutions of differential equations have been
obtained either analytically or numerically based on typically well-behaved
forcing and boundary conditions for well-posed problems. We are changing this
paradigm in a fundamental way by establishing an interface between
probabilistic machine learning and differential equations. We develop
data-driven algorithms for general linear equations using Gaussian process
priors tailored to the corresponding integro-differential operators. The only
observables are scarce noisy multi-fidelity data for the forcing and solution
that are not required to reside on the domain boundary. The resulting
predictive posterior distributions quantify uncertainty and naturally lead to
adaptive solution refinement via active learning. This general framework
circumvents the tyranny of numerical discretization as well as the consistency
and stability issues of time-integration, and is scalable to high-dimensions.Comment: 19 pages, 3 figure
Machine Learning of Space-Fractional Differential Equations
Data-driven discovery of "hidden physics" -- i.e., machine learning of
differential equation models underlying observed data -- has recently been
approached by embedding the discovery problem into a Gaussian Process
regression of spatial data, treating and discovering unknown equation
parameters as hyperparameters of a modified "physics informed" Gaussian Process
kernel. This kernel includes the parametrized differential operators applied to
a prior covariance kernel. We extend this framework to linear space-fractional
differential equations. The methodology is compatible with a wide variety of
fractional operators in and stationary covariance kernels,
including the Matern class, and can optimize the Matern parameter during
training. We provide a user-friendly and feasible way to perform fractional
derivatives of kernels, via a unified set of d-dimensional Fourier integral
formulas amenable to generalized Gauss-Laguerre quadrature.
The implementation of fractional derivatives has several benefits. First, it
allows for discovering fractional-order PDEs for systems characterized by heavy
tails or anomalous diffusion, bypassing the analytical difficulty of fractional
calculus. Data sets exhibiting such features are of increasing prevalence in
physical and financial domains. Second, a single fractional-order archetype
allows for a derivative of arbitrary order to be learned, with the order itself
being a parameter in the regression. This is advantageous even when used for
discovering integer-order equations; the user is not required to assume a
"dictionary" of derivatives of various orders, and directly controls the
parsimony of the models being discovered. We illustrate on several examples,
including fractional-order interpolation of advection-diffusion and modeling
relative stock performance in the S&P 500 with alpha-stable motion via a
fractional diffusion equation.Comment: 26 pages, 10 figures. In v2, a minor change to the formatting of a
handful of references was made in the bibliography; the main text was
unchanged. In v3, minor improvements were made to the exposition; more
details about motivation, examples, optimization, and relation to previous
works were give
Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems
We present a probabilistic deep learning methodology that enables the
construction of predictive data-driven surrogates for stochastic systems.
Leveraging recent advances in variational inference with implicit
distributions, we put forth a statistical inference framework that enables the
end-to-end training of surrogate models on paired input-output observations
that may be stochastic in nature, originate from different information sources
of variable fidelity, or be corrupted by complex noise processes. The resulting
surrogates can accommodate high-dimensional inputs and outputs and are able to
return predictions with quantified uncertainty. The effectiveness our approach
is demonstrated through a series of canonical studies, including the regression
of noisy data, multi-fidelity modeling of stochastic processes, and uncertainty
propagation in high-dimensional dynamical systems.Comment: 37 pages, 11 figures, Submitted to Computational Mechanic
Deep learning of free boundary and Stefan problems
Free boundary problems appear naturally in numerous areas of mathematics,
science and engineering. These problems present a great computational challenge
because they necessitate numerical methods that can yield an accurate
approximation of free boundaries and complex dynamic interfaces. In this work,
we propose a multi-network model based on physics-informed neural networks to
tackle a general class of forward and inverse free boundary problems called
Stefan problems. Specifically, we approximate the unknown solution as well as
any moving boundaries by two deep neural networks. Besides, we formulate a new
type of inverse Stefan problems that aim to reconstruct the solution and free
boundaries directly from sparse and noisy measurements. We demonstrate the
effectiveness of our approach in a series of benchmarks spanning different
types of Stefan problems, and illustrate how the proposed framework can
accurately recover solutions of partial differential equations with moving
boundaries and dynamic interfaces. All code and data accompanying this
manuscript are publicly available at
\url{https://github.com/PredictiveIntelligenceLab/DeepStefan}.Comment: 27 pages, 16 figures, 12 table
Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
Surrogate modeling and uncertainty quantification tasks for PDE systems are
most often considered as supervised learning problems where input and output
data pairs are used for training. The construction of such emulators is by
definition a small data problem which poses challenges to deep learning
approaches that have been developed to operate in the big data regime. Even in
cases where such models have been shown to have good predictive capability in
high dimensions, they fail to address constraints in the data implied by the
PDE model. This paper provides a methodology that incorporates the governing
equations of the physical model in the loss/likelihood functions. The resulting
physics-constrained, deep learning models are trained without any labeled data
(e.g. employing only input data) and provide comparable predictive responses
with data-driven models while obeying the constraints of the problem at hand.
This work employs a convolutional encoder-decoder neural network approach as
well as a conditional flow-based generative model for the solution of PDEs,
surrogate model construction, and uncertainty quantification tasks. The
methodology is posed as a minimization problem of the reverse Kullback-Leibler
(KL) divergence between the model predictive density and the reference
conditional density, where the later is defined as the Boltzmann-Gibbs
distribution at a given inverse temperature with the underlying potential
relating to the PDE system of interest. The generalization capability of these
models to out-of-distribution input is considered. Quantification and
interpretation of the predictive uncertainty is provided for a number of
problems.Comment: 51 pages, 18 figures, submitted to Journal of Computational Physic
Multi-fidelity classification using Gaussian processes: accelerating the prediction of large-scale computational models
Machine learning techniques typically rely on large datasets to create
accurate classifiers. However, there are situations when data is scarce and
expensive to acquire. This is the case of studies that rely on state-of-the-art
computational models which typically take days to run, thus hindering the
potential of machine learning tools. In this work, we present a novel
classifier that takes advantage of lower fidelity models and inexpensive
approximations to predict the binary output of expensive computer simulations.
We postulate an autoregressive model between the different levels of fidelity
with Gaussian process priors. We adopt a fully Bayesian treatment for the
hyper-parameters and use Markov Chain Mont Carlo samplers. We take advantage of
the probabilistic nature of the classifier to implement active learning
strategies. We also introduce a sparse approximation to enhance the ability of
themulti-fidelity classifier to handle large datasets. We test these
multi-fidelity classifiers against their single-fidelity counterpart with
synthetic data, showing a median computational cost reduction of 23% for a
target accuracy of 90%. In an application to cardiac electrophysiology, the
multi-fidelity classifier achieves an F1 score, the harmonic mean of precision
and recall, of 99.6% compared to 74.1% of a single-fidelity classifier when
both are trained with 50 samples. In general, our results show that the
multi-fidelity classifiers outperform their single-fidelity counterpart in
terms of accuracy in all cases. We envision that this new tool will enable
researchers to study classification problems that would otherwise be
prohibitively expensive. Source code is available at
https://github.com/fsahli/MFclass
Gaussian processes meet NeuralODEs: A Bayesian framework for learning the dynamics of partially observed systems from scarce and noisy data
This paper presents a machine learning framework (GP-NODE) for Bayesian
systems identification from partial, noisy and irregular observations of
nonlinear dynamical systems. The proposed method takes advantage of recent
developments in differentiable programming to propagate gradient information
through ordinary differential equation solvers and perform Bayesian inference
with respect to unknown model parameters using Hamiltonian Monte Carlo sampling
and Gaussian Process priors over the observed system states. This allows us to
exploit temporal correlations in the observed data, and efficiently infer
posterior distributions over plausible models with quantified uncertainty.
Moreover, the use of sparsity-promoting priors such as the Finnish Horseshoe
for free model parameters enables the discovery of interpretable and
parsimonious representations for the underlying latent dynamics. A series of
numerical studies is presented to demonstrate the effectiveness of the proposed
GP-NODE method including predator-prey systems, systems biology, and a
50-dimensional human motion dynamical system. Taken together, our findings put
forth a novel, flexible and robust workflow for data-driven model discovery
under uncertainty. All code and data accompanying this manuscript are available
online at \url{https://github.com/PredictiveIntelligenceLab/GP-NODEs}.Comment: 27 pages, 16 figures, 4 tables. arXiv admin note: text overlap with
arXiv:2004.0684